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The average length and average relaxation time of attractors in sequence processing neural 
networks are investigated. The simulation results show that a critical point of a, the loading ratio, 
is found. Below the turning point, the average length is equal to the number of stored patterns; 
conversely, the ratio of length and numbers of stored patterns, grow with an exponential dependence 
exp (Aa). Moreover, we find that the logarithm of average relaxation time is only linearly associated 
with a and the turning point of coupling degree is located for examining robustness of networks. 
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I. INTRODUCTION 

The dynamic behavior of Hopfield model^, a global symmetric coupling neural network, is relatively simple: the 
system relaxes to a fixed point corresponding to a stable patterns^. The latest result of its maximal stored capacity 
as given by Volk"^ is 0.143 ± 0.002. Because the synaptic connections in real biological neural networks have a highly 
degree of asymmetry and a real human idea consists of a set of patterns, many modification of Hopfield model have 
been designed''"^. In general, an asymmetric connection strength may arise to recall a series of patterns or even chaos 
behavior, and cannot be studied by conventional statistical methods because it disobeys detailed balance. 

In this paper, we study the behavior of attractors in a sequence processing model^, a fully connected Ising spin 
model, through numerical simulation with deterministic parallel dynamics. Moreover, the robustness is examined by 
locating critical coupling degrees since the loss of synaptic connections in the human brain may occur because of brain 
damage^. 

II. MODEL DEFINITION 

We consider a sequence processing neural network model described by the following equations'*"^ 

s,(f+l) = sgn|^^JijSj(t)j (1) 

where Ising spin Si{t) G { — 1, 1} represents the firing state or resting state and sgn (x) is updating function corre- 
sponding with signum function which if x > 0, we get 1; if a; < 0, we get —1. The neurons evolve their states 
simultaneously with deterministic parallel dynamics and the time step is 1 in all our work. 
The synaptic connection matrix J can be given by'*'* 

•^'^■ = ^E^''A'^'^" (2) 
ti=i 

for a diluted model consisting of N neurons. The q vectors f = {'Cfi j • ■ • > Sat} ^ {"IjI}^ randomly and 
independently stored patterns in neural networks. It means the stored patterns are organized in one g-period cycle. 
It can be introduced by definition ^1^^ — S^j in (2). The dilution factors Cy = or 1 are independent identically 
distributed random variables. They are selected by 

if z < d, then cy = 1; else cy = (3) 

where z G [0, 1] is a random number and d e [0, 1] is the coupling degree of networks. For Cy = 1 or d = 1, the model 
restore to a standard (Hopfield- Hebb) sequence processing model. 
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Comparing with standard Hopficld model, the synapses of equation (2) arc insufficient for the generation of stable 
fixed point in phase space, since they combine pattern jj, with pattern fi + 1. Nevertheless, the transition from one 
pattern to another is so fast that it possible get into a stationary limit cycle for the modulo-g checking of fi in 



III. THE PERFORMANCE PARAMETERS 

For measuring the retrieving ability of this model, in a similar way to the definition in Hopfield models, the overlaps 
of state s (t) and the stored patterns with the initial arbitrary pattern s (0) £ { — 1, Ij'^is defined by 

1 ^ 

For example, in 100 neurons with 10 stored patterns in a cycle, we obtained a pattern sequence as8— >7^6^5^ 
4^3^2^1^10^9^8^7-- - (see Fig. la). It is easy to find that the series is equal to a cycle of stored 
patterns with 10 patterns for different retrieving times. 

There are another two important parameters for describing long time behaviors of networks: the cycle size p and 
the relaxation time r. They are defined by®~^^ 

p = min [s (t + n) = s (t)] (5) 



r = min [s (p + n) = s (n)] (6) 

n 

Here, p is the periodicity of attractors and r is the time the system converging into the p-period cycles. 

In fact, the values of p and r are dependent upon the initial patterns s (0) and the coupling matrix J or the stored 
patterns So, for a given value p, we can choice many connection matrices through randomly generating a large 
number of stored patterns. Moreover, the average size of cycles and the average relaxation time can be defined by^'^° 

M 



in which p (n) and r (n) are defined by (5) and (6) for the n-th sample. 

Now, viewing the stored patterns series as a whole, our interest is the overlaps between s (t) and the stored cycle, 
not with a simple pattern in the stored series. As a special case, only for p = q, we introduced the average cycle 
overlaps 

1 

{m){T) = -^maxmi^iTq-i + l), i^=l,...,q, T=l,2,... (9) 

^ i=l 

in which T is the refreshed time scale. Figure lb shows that the system plotted in Figure la is successful in forming 
(m) = 1 for g = 10, but there is a very small error with (m) = 0.963 for q = 20. 



IV. SIMULATION RESULTS 

For searching the features of period and relaxation time of attractors, we set the Cy = 1 corresponding to the 
standard sequence processing networks in fig. 2 to 4. We focus attention on the relations (p) vs. a and (r) vs. a in 
which the loading ratio a is defined by q = aN . /From comparing the value of M, the numbers of samples, given by 
1000 or 200, we find simulation results of {p) vs. a have only a very small error. Considering our computer device, 
the value of N ranges from 50 to 150 and the value of M is 200 in the following calculation. 
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Obviously there is a turning point ac dividing the curve of (p) vs. a into two parts (see Fig. 2). In the first part 
with a < ac, we find stable behaviors of cycles with (p) « aN equal to q, the muiiber of stored patterns. Beyond this 
point, the curve gradually changes into an exponential dependence of (p) / (aN) (x exp (Aa). Here, A is a constant in 
the range from 8.26 to 46.74 for iV = 50 to 150. The value of (p) increases so drastically that no results of a > 0.35 
are plotted. Additionally, the ac are in the range 0.13 to 0.17, and respectively the formation ratio k/M = 0.80 to 
0.85, where k is the number of g-period attractors with (m) > 0.90 and M = 200 is the number of samples (see Fig. 
3). 

Figure 4 shows the average relaxation times in fully coupled networks. Similarly, an exponential dependence 
(r) oc exp {Ba) was observed with B = 12.01, 18.98 and 29.07 for TV = 50, 80 and 120, respectively. 

In order to investigate the robustness of networks within the limits of ac, simulation with diluted connection have 
been carried out through changing the values of coupling degree d or setting Cjj belong to {0, 1} randomly. In Fig. 
5, in the same way as figure 2, it is easy to see that there exists a critical point dc = 0.15, 0.35 and 0.60 for a = 0.05, 
0.10 and 0.15. For stronger dilution, the average periodicity grows towards a plateau higher than aN. 

In conclusion, for small systems with spin neurons, it can be seen that the ratio of the average size of attractors (p) 
and the numbers of stored patterns is 1 as long as the loading ratio a > ac, and then it increases strongly following 
an exponential law exp (Aa) . The more the system size increases, the more the ratio grows. From analyzing the 
average cycle overlaps (m) (Fig. 3), there is a narrow quickly decreasing region near the turning point of periodicity. 
The maximal stored capacity^ at as = 0.23 for A'^ = 100, corresponding to k/M = 0.66, is much larger than at 
as = 0.32 ( when k/M = 0.005 ), under the condition that (m) > 0.90. Moreover, the average relaxation time 
depends exponentially on the loading ratio and it raises faster with larger systems. 

Additionally, through the simulation of the coupling degree d in networks, it is clear that there is a narrow transition 
region between small and large (p) /{aN) centered on an inflection point dc- Moreover, the larger loading ratio 
corresponding to the larger value of dc means its robustness of stored information is more unstable. Beyond our 
expectation, one result is a stable large attractor for highly diluted systems, another is that possibly dc depends only 
on a for several different system sizes. This needs more detailed work in the future. 

Above all, it is easy to see, not as in the asymmetric model given by Gutfreund et al., that the asymmetric factor is 
a major parameter^" that the loading ratio a is a very important factor in sequence processing networks. In fact, 
this difference from the definition of asymmetric connection strength by Gutfreund et al. 
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Fig. 1. (a) pattern overlaps and updating time for N = 100 with 10 stored patterns in a cycles, (b) the average 
cycle overlaps plotted against the refreshed time scale T. 

Fig. 2. logarithm of {p) / {aN) against the loading ratio a for four different system size N. 

Fig. 3. the formation ratio k/M for (m) > 0.90 and M = 200 against a for a < ac and = 50 to 150. 

Fig. 4. logarithm of (r) against the loading ratio a for three different system size. 

Fig.5. logarithm of {p) / {aN) against the coupling degree d for loading ratio a for A'' = 100. 
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Fig. 1a 
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Fig. 2 
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Fig. 5 



